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Executive Summary 


Recruitment is a key challenge for researchers conducting any large school-based study. 
Random assignment studies, however, can pose greater challenges than others because of 
the degree of control that researchers often require. Teachers and schools are accustomed to 
autonomy and reluctant to give up control, even in part. 

Control is needed not only over the condition participants receive, but also over how 
the intervention is implemented, and may include restrictions in other areas of school 
and classroom functioning. Further, to have sufficient statistical power to detect at least 
medium-sized effects (effect sizes of .20 standard deviations or larger) and allow for attrition, 
researchers typically need to obtain samples of 30-60 teachers or schools. 

We report here on our experiences in recruiting participants for random assignment 
experiments in public primary and secondary schools, often across midsize and large school 
districts. Our perspective is based on over twenty current and completed randomized 
controlled trials (RCT) in K-12 school settings conducted by SRI International, including 
studies of educational technology, literacy, mathematics, science, instructional materials, 
teacher professional development, and student behavioral supports, following What Works 
Clearinghouse standards for RCTs. 1 A list of such studies appears in the Appendix, with 
an abstract for each study. Considering our experience across these studies, we reflect on 
how we approached the recruitment problem and what worked during our efforts. In general, 
recruitment rarely goes according to plan, often runs into unexpected challenges, and typically 
requires more resources than originally anticipated. Project leaders are often called upon to 
respond creatively to these challenges. As we examined our own projects as case studies of 
how project leaders manage the problems, patterns of successful projects emerged. Given 
the paucity of information on the challenge of recruiting, we were unable to apply a formal 
literature review or research method to organizing our effort, which is essentially a synthetic 
analysis of common patterns in multiple cases. We acknowledge limitations of this method. We 
aim to stimulate a broader conversation about recruitment, rather than to definitively answer 
questions. At the same time, the preliminary insights reported here may be of some value to 
others in the field who face similar challenges. 


1 http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v3_0_draft_standards_handbook.pdf 
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Our comments are organized in six topics corresponding to the aspects of an overall 
recruitment process: 

1 . Study design 

2. Intervention packaging 

3. Planning a recruitment process 

4. Designing recruitment messages 

5. Running a recruitment campaign 

6. After recruitment. 

Although we describe these topics in a linear manner for clarity, in fact the recruitment process 
can have parallelism and include iterative cycles. Further, the process of recruitment usually 
requires attention to multiple levels (e.g., districts, schools, teachers) simultaneously because 
individual participants are nested within classrooms, schools, and districts and buy-in must 
occur at each level. The conclusion summarizes our recommended phases of a recruitment 
process and suggests that research proposals and peer review processes pay more attention 
to the details of recruitment. The following page summarizes our recommendations. 
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SRI Education Recruitment Recommendations 


Design: 

Key recruitment considerations during design of a 
study are 

• Unit of randomization 

• Sampling from a population 

• Finding a balance between a desired intensity of 
intervention and ways to contain the intervention to 
only a portion of a teachers’ work life 

• Reducing data collection burden 

• Considering using multiple cohorts 

• Design options that give all participants the 
intervention at some point. 

Intervention Packaging: 

It is important to package the intervention to ensure a 
strong fit with the structure and policies of schools or 
districts and needs of teachers while still maintaining 
the core of the intervention and meeting the 
researchers’ goals for the study. Considerations may 
include: 

• Managing the productivity issues associated with 
participating 

• Highlighting alignment of participation with broader 
goals 

• Designing financial incentives to offset perceived 
extra costs. 

Planning the Recruitment Process: 

During the planning of the recruitment process, we 
find it worthwhile to 

• Consider the multiple levels of K-12 school systems 

• Get an early start to recruitment in tandem with a 
well-articulated plan 

• Leverage local connections 

• Tap in to professional networks. 


Designing Recruitment Messages: 

We have found the following valuable in planning 

recruiting messages: 

• Communicate the value of the study in ways that 
will resonate with recruits, convey a clear image of 
the program, highlight costs and benefits, describe 
layers of reasons to participate, identify requirements 
and expectations, and address typical concerns 
raised by participants 

• Develop a consistent approach to talking about 
random assignment 

• Support consistent messaging in a variety of useful 
formats, including handouts, presentations and 
demonstrations, a website, and video 

• Create an appropriate level of commitment by 
requiring application to the study. 

Campaign: 

During a campaign, important elements include: 

• Weekly team conference calls 

• A shared recruitment database 

• Following a sales pipeline metaphor 

• Operating on both broadcast and interpersonal 
fronts simultaneously 

• Assigning interested participants to a consistent 
point of contact 

• Remaining flexible and adaptive in face of emerging 
recruitment experiences 

• Maintaining team spirit and motivation through 
dispiriting times. 

After Recruiting: 

After recruiting, it important to follow through on 

details such as: 

• Prompt notification of participants 

• Inducting the participants into the study 

• Building rapport and relationships 
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Study Design 



In our experience, successful recruitment begins while the study is being designed because 
the design informs all aspects of the recruitment process. During the design stage, researchers 
need to balance factors that could improve the quality and value of their study with what is being 
asked of participants, which is a central issue in how easy or difficult recruitment will be. Issues 
to consider are the unit of randomization, sampling, balancing intervention intensity with the 
burden on participants, and data collection burden. Many of these, if not all, are decisions that 
should have been made at the proposal stage. After discussing these issues, we describe two 
design options that can sometimes ease recruitment challenges by enabling all study participants 
to receive a desired intervention. Other non-design-dependent factors such as the location of 
the study, the structure of district and school decision-making, and other ongoing initiatives 
also affect the recruitment process, and those are discussed later. Please note that we will 
sometimes uses the word “program” and sometimes the word “intervention” in describing what 
the experiment is manipulating; in different study contexts, one or the other term fits best. 

Other framing issues such as ethical and legal considerations may rise from the beginning 
of a project, during the recruitment stage. The top ones would be voluntariness, privacy, and 
confidentiality. Additionally, there are also the ethical issues around equity of participation, 
making sure that we, researchers, are providing something of use and value to the participants 
during the studies, without doing any harm. Some school districts also have human subjects 
policies that need explicit compliance. Although these issues are highly important, they go 
beyond the scope of this white paper and we will not discuss them in further detail. 


Unit of Randomization 

The research design must specify the unit of random assignment— most likely students, 
classes, teachers, or schools or, less commonly, districts. There are design trade-offs ranging 
from the size of the study (if randomization is at a higher level) to contamination and crossovers 
(if randomization is at a lower level) to implications for how to manage communications during 
recruitment. 

If the data are clustered (because of the nested nature of students within teachers within 
schools), then the highest level of the clustering hierarchy drives statistical power. Therefore, 
studies with randomization at higher levels (e.g., schools or districts) often have to be at 
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larger scale (and therefore have larger recruitment efforts) than studies with randomization 
at lower levels (e.g., students). On the other hand, if students or teachers will be randomly 
assigned, contamination is a concern because students or teachers within the same school are 
somewhat likely to share resources regardless of which condition they have been assigned to. 
This problem can be most acute if randomization is done at the classroom level, with teachers 
giving both treatment and control conditions to different classes. Before deciding to randomize 
at a lower level, researchers need to pay special attention to whether their intervention can 
be rolled out in such a way that these risks can be contained. Special procedures to avoid 
contamination may also make recruitment harder because asking teachers or students to 
avoid sharing may violate school norms. When schools are randomized, procedures to avoid 
contamination are less of a recruitment concern because less sharing occurs between schools. 

In addition to decisions that affect the scale and quality of the study, design decisions have 
implications for how the study may be perceived and, therefore, the likely challenges of the 
recruitment effort. If students will be randomly assigned, a challenge for recruitment will be 
obtaining school buy-in to allow researchers to assign students to specific teachers, specific 
materials, or specific instructional programs. Such assignments are typically controlled by 
schools and are subject to parents’ involvement. Concerns about fairness often come up 
during recruitment; people are reluctant to volunteer for situations they believe are not fair. 

Such concerns may be greater when random assignment is at lower levels as participants in 
treatment and control groups may be aware of each others’ condition, (e.g., if some teachers 
or students in the same school receive something that others do not). If some schools will be 
in a business-as-usual condition and others will receive something new, however, fairness 
concerns can arise even when random assignment is at higher levels. Overall, regardless of the 
level of random assignment, we have found it important to build buy-in across all levels (e.g., 
students and their parents, teachers, school principals and district leaders). 


Designing a Sampling Strategy 

It goes without saying that larger samples are more difficult to recruit than smaller ones. A related 
consideration is allowing for attrition; if participants are likely to leave the study after random 
assignment, recruiting more participants will be necessary. Consequently, care is needed during 
the design of large-sample studies to keep attrition low and sample sizes reasonable. 

Randomly sampling participants from a population during recruitment is extremely rare in 
education studies (e.g., choosing which schools in Texas to recruit by random selection from a 
list of all schools). In practice, a sample population is recruited purposefully but non-randomly 
(with random assignment occurring among the units within that sample). Important technical 
issues include considering whether there are subpopulations for which you might want to 
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estimate treatment effects, and/or if you might want to generalize the findings to particular 
populations. These come into play in identifying the strategy for purposeful sampling. Careful 
consideration of such issues can pay dividends in terms of the generalizability of findings . 2 

While making these choices, study designers consider what characteristics of a broader 
population should be represented in a study sample to support desired generalization. Choices 
about who to sample can also affect the ease or difficulty of recruitment. For instance, in a 
recruitment effort conducted by the SimCalc research team in Texas, regional differences were 
considered important. Therefore, the researchers decided to recruit in particular regions including 
two urban areas (Austin and Ft Worth), the remote western region of Texas, and the unique 
communities of the Rio Grande Valley near Mexico. The team also wanted to include Houston or 
Dallas, which have larger African American populations and very large school districts, but both 
declined to participate. In recruiting for the ASSISTments study in Maine, regional differences 
were less important; the researchers sought schools of different sizes (public school sizes in 
Maine vary widely) and different levels of mathematics achievement. How the target for sampling 
is defined can make the recruitment more or less challenging. In Texas, a very large state, it 
was easier to concentrate on defined regions than on the whole state; in Maine, the focus on 
school size and prior achievement reduced the need to include very remote regions of the state, 
which would have imposed travel burdens and were not expected to respond to the intervention 
differently from other schools of similar size and similar prior mathematics achievement. 

Sometimes, especially in efficacy (as opposed to effectiveness) studies, it may be important 
to consider selecting a sample that is conducive to implementing a program with fidelity. For 
example, the purpose of the Evaluation of the College-Ready Writer’s Program is to assess the 
efficacy of a new program the National Writing Project developed in response to the Common 
Core State Standards. For this recruitment effort, the researchers invited Local Writing Project 
sites to apply to participate in the study and selected those that had experience offering other 
similar professional development programs to increase the likely implementation fidelity. 


2 Tipton, E., Hedges, L. V., Vaden-Kiernan, M., Borman, G. D., Sullivan, K., and Caverly, S. (2014). Sample Selection in Ran- 
domized Experiments: A New Method Using Propensity Score Stratified Sampling. Journal of Research on Educational 
Effectiveness 7:114-35. 
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Balancing Duration, Intensity, Impact and Participant Burden 

Often researchers expect to see a relationship between the duration and intensity of the 
intervention and the magnitude of expected impacts. A longer duration and greater intensity 
intervention may increase expected impacts, thereby reducing the necessary sample size. 
However, it clearly is far more difficult to recruit (and retain) teachers for an experiment that 
lasts longer and involves more substantial changes in work practice than it is to recruit 
college sophomores for a lab study that lasts 2 hours. Further, it may be more practical or 
participants to engage in a high intensity of a new practice over a shorter period of time. In the 
design phase, researchers must carefully consider the likely benefits of higher duration and 
intensity and balance them with the increased demands they place on participants and the 
corresponding challenges in recruiting and retaining them. 

In all the example studies that we considered, teachers who were assigned to the treatment 
condition were expected to make substantial changes to their work practice (e.g., switch to 
an entirely new curriculum or change the way they assign and review homework). In these 
cases, assignment to condition constrained teachers’ choices of how they did their jobs, and 
this is an important burden for researchers to consider. For studies in which the control group 
is a business-as-usual condition, the control group may face fewer constraints, which may 
ultimately be perceived as a benefit to the less enthusiastic participants who are assigned to it. 

For long-term, more intrusive experiments, we have found it valuable to consider how the 
intensity of the intervention might be faithfully implemented within reasonable bounds for the 
implementing teacher. For the SimCalc research, the team wanted teachers to adopt a fairly 
radical change in how they taught the mathematics topic of “proportional reasoning” using a 
novel technology, but the team was willing to limit the duration of the change. Consequently, a 
replacement unit model was used, in which the intervention was positioned as a one-for-one 
replacement for the materials usually used to teach a particular unit for 2-3 weeks. Having 
a limited duration reduced teachers’ need to make a long-term commitment to change in 
instruction and made recruitment easier, but the length of time was long enough for measuring 
substantial effects and observing realistic implementation difficulties. For the ASSISTments 
research, the intervention developer focused on a particular instructional practice, homework. 
Here again, a fairly intensive change was sought but only to a slice of a teacher’s overall daily 
routine, homework assignment and review, leaving other classroom practices untouched. 

Of the studies discussed here, the Reasoning Mind efficacy study is the most ambitious. The 
researchers are asking teachers to switch from the curricular materials they use to teach fifth- 
grade mathematics to technology-based, integrated instruction and practice materials for an 
entire school year. The rationale for studying the full package of Reasoning Mind was that the 
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likelihood of the program achieving its desired impacts through implementing all the integrated 
components outweighed what was being asked of participants and the recruiting challenges 
the design posed. Further, schools are adopting new mathematics curricula to comply with the 
Common Core standards; the offer to use a completely new curriculum at no cost was received 
favorably by schools. We imagine that schools will be less receptive in a few years, after they 
have already committed to the selection and purchase of particular materials for the Common 
Core. (Indeed, another concern we encountered is that schools worry that they may later be 
charged for materials they initially receive for free.) 

One concern about studies of program effects is that they are often conducted before teachers 
and students have had sufficient time to master the use of the new program; measuring effects 
while participants are still coming up to speed on a program can result in under-measurement 
of the eventual effect. RCTs may be especially prone to this because it is difficult to require 
participants to “stay in condition” for a longer period time. However, novice implementers may 
in turn reduce the quality of implementation and undermine the interpretation of the study. 

This was an important consideration for the researchers on the Project-Based Inquiry Science 
Efficacy study, who recruited schools that were new to the curricular intervention. The study 
team knew from prior research that typically two rounds of implementation are required for 
teachers to become familiar with new instructional materials and the pedagogy underlying 
them. For this reason, assignment to condition for schools and teachers had to last 2 years. 

In the ASSISTments and Reasoning Mind experiments, the intervention developers similarly 
wanted a full warm-up year for teachers to get used to the technology platform and master 
the new pedagogy, so the measurement of the main contrast was designed to occur in a 
second school year with a different cohort of students. Therefore, as in the Project-Based 
Inquiry Science Efficacy study, assignment to condition for teachers or schools had to last 2 
years. The researchers concluded that the benefits of measuring impacts during more mature 
implementation outweighed the challenges of conducting a longer study. To help ease the burden 
of this design, the research team reduced the amount of data collected during the warm-up year. 


Demands of Data Collection 

An RCT addresses the extent of measurable impact. When a study can collect valid measures 
of implementation, key mediators and moderators, and multiple outcome measures, it can also 
support program and theory development and also answer the basic questions of whether, 
how, and for whom the intervention works. The National Writing Project’s SEED (Supporting 
Effective Educator Development) grant provided elementary school teachers with 45 hours of 
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professional development on writing instruction to support them in shifting their instructional 
practices to align with Common Core State Standards. By measuring implementation, proximal 
outcomes (impacts on teacher practice), and distal outcomes (impacts on student writing), the 
evaluation was able to trace the causal chain from program to teachers to students. However, 
this required substantially more data collection in the form of teacher participation and teacher 
surveys and interviews, in addition to pre- and post-testing of students, than if the study 
had narrower goals. The design period is critical for balancing the potential breadth of the 
study with participant burden because as for duration and intensity, the more that is asked of 
participants in terms of data collection, the greater the challenge in recruiting them and the 
higher the risk of attrition during the study. 


Using a Multiple-Cohort Approach to Make Recruitment 
More Manageable 

Doing recruitment all at once is difficult, and sometimes research designs can be adjusted to 
spread recruitment across multiple years. One such design option is staggering recruitment. In 
the SimCalc research, the team decided to recruit for the seventh- and eighth-grade studies 
separately in different years, staggering individual studies over time. With the lessons learned 
from seventh-grade recruitment, eighth-grade recruitment was much easier. In the ASSISTments 
study, the research team planned a two-cohort design. The first pool of schools was recruited 
and randomly assigned in the first year. The second pool of schools was recruited and randomly 
assigned in the second year, staggering the experiment in two cohorts. Assuming no major 
contextual changes in the 2 years (an assumption that should be carefully considered), the data 
collected from both cohorts can be combined to reach the desired sample size, and a single 
analysis can be conducted to estimate the overall impact of the intervention. 


Designs Where Every Study Participant Receives the Program 

Another design consideration is to make sure that every participant receives something 
perceived to be of value during the experiment (if plausible) or eventually. Financial incentives 
(considered later) help but alone are often not enough to secure teachers’ participation. Much 
more typically, schools and teachers see the offered program as addressing an existing need 
and want the opportunity to try it. To accommodate this, in several studies (e.g., the SimCalc 
seventh-grade study, ASSISTments, the SEED Evaluation, the Evaluation of the College-Ready 
Writers Program, and the efficacy study of Project-Based Inquiry Science), we used a delayed 
treatment design. Schools or teachers were randomly assigned to receive the intervention 
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immediately or to wait for some time to gain access to it. From the experimental perspective, 
those who were waiting maintained business-as-usual practices that would provide the needed 
contrast. This proved to be especially attractive to the 42 schools participating in the Project- 
Based Inquiry Science Efficacy study. The schools were all in one large urban school district 
and randomly assigned either to the treatment condition that received a new comprehensive 
curriculum with teacher materials and accompanying science materials for students or to the 
business-as-usual control condition. District leaders were initially hesitant to join an RCT that 
they assumed would benefit some participating schools but not others. Once introduced to the 
delayed treatment design, they viewed it as a fair approach that would make participation of 
value to all schools. 

In the SimCalc eighth-grade study, we took a different approach, giving teachers one of two 
equally valuable offerings. In the replacement unit design, the investigators believed they could 
safely assume that the learning outcomes of the two instructional units on linear functions 
and data analysis were relatively independent of each other, and highly desirable professional 
development was available for both topics. The experiment was thus designed to randomly 
assign teachers to one or the other replacement unit and associated professional development. 
Teachers participating in the data analysis intervention served as the control group, and the 
other group of teachers implementing the linear functions replacement unit was considered the 
treatment group. 

Yet a third choice was made in the Reasoning Mind study. Here, the belief was that schools 
would find the offer of 2 years of free access to the Reasoning Mind curricular product valuable 
but might not want to participate at all if they could end up in a control group and have to wait 
2 years to use the product. The focus of this study was fifth grade. We decided to randomize 
schools to have free access to either the fifth-grade (treatment condition) or the second- 
grade (control condition) version of the product. This way, all schools in the study would get 
something they perceived to be very valuable right away. The availability of the second-grade 
product in the control group schools would have no impact on the learning outcome in the fifth 
grade during the two experiment years. 

From high-level research design perspective, the considerations we discuss above, such as 
duration, intensity, multiple cohort approach, or where each participant receives the program all 
fall into the broad and critical topic, that is, the design of counterfactual. 
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Intervention Packaging 



We posited that one factor that may make recruitment more challenging in RCTs than in other 
study types is researchers’ need to control the intervention received by each participant. 
Because participants will have less control, reducing the burden they experience is important, 
and this falls to the researchers to understand and address. 

Consequently, running an experiment in schools may involve packaging an intervention to fit 
local circumstances and to make implementation and research less burdensome. This does 
not mean redesigning the intervention. In fact, this packaging must retain the core features 
of the program to be tested; otherwise, the study will not have external validity. Yet from our 
experience, it would be a mistake to consider an intervention fixed and unchangeable. Three 
key issues in intervention packaging — productivity, alignment, and extrinsic incentives— must 
be addressed in an iterative cycle in designing the study and the recruitment approach. At 
the heart of all three is identifying the incentives— intrinsic and extrinsic— that are critical to 
recruiting participants. To accomplish this goal, researchers need to think from participants’ 
perspective and offer something that is truly valuable, minimally intrusive and aligned with their 
needs. Sometimes the program developers may resist talking about packaging the intervention, 
but typically we have found that once they understand the extra burden to participants 
for being part of a study, they recognize the need to consider how to ease the effort for 
participants. Overall, we have found that modest changes around the edges of the intervention 
can make recruitment and implementation much easier and can also lead to a higher quality 
research design. 


Productivity 

One key issue in introducing a new intervention, especially when data collection will requires 
additional participant time, is that teachers have a set amount of time to accomplish all their 
normal teaching responsibilities. Although researchers can sometimes offer teachers nominal 
compensation for participation, they cannot create more time for teachers to do the additional 
work. A key task in intervention packaging therefore is for researchers to think of strategies to 
enable teachers to ramp up on the new intervention while also fulfilling their regular teaching 
duties and participating in the required data collection. 
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In the ASSISTments study, it was fairly easy to make a good case to teachers that the 
technology could reduce work compared to their typical ways of assigning and grading 
homework. (This is unusual; many technologies introduced into schools tend to make more 
work for teachers in the short term if not long term as well.) The ASSISTments platform is web 
based, and every individual student already has access to a laptop because of Maine’s one- 
on-one computing initiative. The use of ASSISTments can also open up class time for teacher 
instruction through a quicker start to class and more focused homework reviews because 
students receive immediate feedback on their homework and teachers have the opportunity 
to review a report on students’ homework performance before or at the beginning of class. 
Although we knew these potential productivity gains would be attractive to teachers, we were 
worried that the amount of time it would take to enter homework into the ASSISTments system 
could be perceived as burdensome. Thus, the developers collected every textbook used by 
participating schools and entered all the possible homework problems for the teachers, saving 
them weeks of potential work. 

The SimCalc studies did not have this inherent productivity benefit, and teachers needed 
to take additional time to install the software and arrange for access to computers in a lab 
or laptop carts. Therefore, we directed much effort to planning logistics, making it as easy 
as possible for teachers to do each step necessary for conducting the study. This included 
providing the teachers with a box containing everything they would need for implementation 
and data collection (e.g., workbooks, packing tape and prepaid mailing labels for returning data 
to researchers), just-in-time supports, and coherent instructions presented through multiple 
media. Every step of the process was considered, and the team worked with the developer to 
package materials for clarity and ease of use. 

For the Ready to Learn Prekindergarten Transmedia Mathematics study, support for the teachers 
was considered essential. It was understood that the integration of new technology along with 
the curricular supplement would certainly require a teacher’s time and an added commitment to 
learning about a technology product the teachers most likely had never encountered. Therefore, 
the study design included on-demand technical support and ongoing coaching. 

These examples show how researchers worked with intervention developers to increase 
teachers’ productivity (or at least minimize the decrease in productivity) during their initial 
program implementation. Such adaptations are common in efficacy studies where programs 
are implemented under supportive conditions. Reporting on studies in which teacher burden 
was meaningfully reduced should be clear about how the intervention differed from typical use 
scenarios, for example, to support others in making appropriate generalizations. 
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Alignment 

A related and highly important issue during intervention packaging is alignment with broader 
mandates within districts or states. Schools and teachers choose to participate in studies that 
will enable them to be better aligned with the directions their districts or states are heading and 
avoid studies when they sense a conflict. Designers often have choices in framing a study that 
can heighten perceived alignment. For example, “SimCalc” originally meant “simulations for 
calculus,” which was appropriate for its original higher education focus but not for the intended 
middle school mathematics experiments. Consequently, the study team never told participants 
that “Calc” meant calculus for fear they would see it as misaligned with their teaching goals. 

The team went to great lengths to focus the intervention on the middle school topic of “rate and 
proportionality,” a topic that most teachers would recognize as very important, that was clearly 
called for in the Texas state standards, but that was known to be hard to teach. Indeed, the team 
chose to conduct the study in Texas in part on the basis of the quality of the alignment of the 
intervention with the statewide goals. Highlighting this alignment made recruitment much easier. 

Likewise, for ASSISTments, the team could have highlighted “intelligent tutoring” as an 
advanced technology, but states do not directly call for intelligent tutoring - and thus this 
might have been seen by schools as potentially irrelevant to their mission. However, Maine 
has had a statewide one-on-one computing program for almost 10 years under which every 
middle school student receives a laptop and may take it home for homework. Despite the 
availability of the devices for homework, the state does not provide any software to make 
homework on the laptops easier or more valuable. Hence, the team communicated the purpose 
of the ASSISTments study as improving the effectiveness of homework practice and making 
good use of student laptops. The focus on a state-specific issue or challenge can be more 
compelling than a focus on a more general advance. 


Financial and Other Potential Incentives 

Our philosophy across all studies has been that financial incentives offset additional costs that 
teachers would otherwise bear for their participation but are not a primary reason to participate. 
Through extensive experience conducting research across a range of designs, we have come 
to see that the intrinsic value of participating in research (e.g., to voice an opinion, to try a 
new strategy that may improve performance, to participate in a new program with colleagues, 
or to get help with an established need) are a much stronger incentive that any financial 
compensation researchers can typically afford. Instead of thinking of financial compensation 
as an enticement, we use financial incentives to counter the disadvantages of participating. 
Accordingly, we have tended to peg the incentive amounts to the approximate number of hours 
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teachers will spend doing extra work for the research (e.g., completing surveys, distributing 
materials, participating in professional development) multiplied by a reasonable hourly rate 
in the teachers’ home state. For example, teachers are paid $25 per hour for participating 
in ASSISTments training and Reasoning Mind offers up to $1 ,000 for teachers who fulfill all 
professional development and certification requirements. 

An important consideration in designing financial incentives is to make sure incentives 
compensate those who are shouldering the burden of participation. In studies where the initial 
recruitment occurs at the district level, districts may receive a financial incentive to counter 
the potentially negative perceptions of being assigned to the control group. A typical example 
is the Evaluation of the College-Ready Writers Program, in which treatment districts received 
free professional development and control districts received grants of $5,000 per year that they 
could use for any legitimate educational purpose other than writing professional development 
for grades 7-10. While this might offset district concerns about being in the control group, 
paving the way for recruitment, teachers bear the time burden for study participation. As a 
result, the study provided incentives to them as well, tied to completion of major data collection 
activities, to promote effective data collection. Another twist on the principle of using financial 
incentives to offset the costs of someone’s time is in studies requiring districts to provide 
relatively extensive extant data for analyses (or some other type of personnel time devoted to 
supporting the study). In such cases, incentives may go to the district office to support staff 
time necessary to fulfill the requests. 
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Planning a Recruitment Process 


Recruitment is a process that unfolds in several stages over an extended period of time and 
requires the coordinated action of multiple staff members. As such, a good plan for the 
recruitment process is essential before actual recruiting begins. 


Consider Multiple Layers of the K-12 System 

The first factor to consider in recruitment is the number of layers of the educational structure that 
need to be addressed in the process. In a typical K-12 system, students are organized into classes, 
taught by teachers who are members of a school faculty, and then multiple schools often make 
up a district. Here, we consider only districts, schools, and teachers (and set aside the somewhat 
different issue of obtaining parental consent for student participation). Securing participation at one 
level may help build support at other levels but is not a guarantee of overall buy-in. 

• Districts — Attaining district-level support may be the first step in the rollout of the 
recruitment process because it enables the research team to reach many schools. 3 District 
approval, however, especially if it is not much more than consent for school and teacher 
recruitment to proceed, may produce little commitment from other levels to do the work 
required by the study. 

• Schools — Principals may exercise substantial discretion and authority within their schools. 
A supportive principal can facilitate teachers trying new strategies or programs and can 
address parental concerns if they arise. A supportive principal can also be an asset later 
in the study, for example, if problems occur with teachers’ compliance to data collection 
requirements. However, principal support is not always indicative of teacher buy-in. 

• Teachers — Teacher-level recruitment has the benefit of securing the commitment of those 
who will probably do the bulk of the work in the study. Given their place in hierarchy, 
however, teachers may not have the authority to ensure supportive conditions for 


3 Districts sometimes have specific polices or requirements regarding IRB, research data collection and data use, or 
resource allocation. In such cases, it would be most efficient to learn early on about such polices or requirements 
to ensure the recruitment process (and later on the study) goes smoothly. 
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implementation (e.g., time on the school calendar or substitute teachers so that teachers 
to attend required professional development). 

Because the interests of actors at each level are not always aligned yet all can play key roles in 
ensuring study success, multiple parallel strategies at different levels need to be implemented 
and coordinated. 

Once the levels required for a particular study are understood, the next step is determine where 
to start. Knowledge of district and school decision-making patterns, which differ by state and 
context, is crucial for making a sound decision. For example, in Maine and West Virginia, where 
the ASSISTments and Reasoning Mind study teams recruited, teachers have a considerable 
voice in instructional decisions, so the teams approached teachers first and then quickly 
included their principals and district superintendents. However, in Texas where the SimCalc 
team recruited, the law requires obtaining district superintendents’ approval before recruiting 
on the teacher level. The Project-Based Inquiry Science Efficacy study team sought to recruit 
schools within very large urban districts that had centralized decision-making processes. In 
that case, working top down was the only viable and perhaps the most efficient and effective 
approach. In other projects, a dual recruitment approach where the team notifies schools and 
generates interest while simultaneously reaching out to the districts with research applications 
has proven beneficial. At times, starting with the schools first has created the momentum for 
securing buy-in so that district approval, if needed, is more of a formality. 


Determine When to Commence Recruiting 

The second consideration is when to begin recruitment. We have typically worked backward 
from when teachers or schools need to be assigned to conditions. Often the first professional 
development workshop is held in the summer before the school year when the intervention 
needs to be implemented. Teachers therefore need to be randomly assigned and notified 
of their conditions by early May so they can be prepared to attend the summer workshop. 
Recruiting a large sample of participating schools or teachers may take up to a year depending 
on such factors as the required sample size, the intervention, and nature of the experiment. 
Thus, the latest time we would recommend for beginning the process is January (4-5 months 
in advance). For the first cohort in the ASSISTments study, the project started in spring (which 
is unusual) because of specific situations in the state of Maine. This left only 2 months for 
recruitment, which was a struggle. We had too few schools in the first cohort but made up 
for it by recruiting more in the second cohort through a 1 -year-long recruitment effort. For the 
Reasoning Mind study, we began in October and allowed 8 months for recruitment. 
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Leverage Local Connections 

The frontline recruiters must be chosen wisely. We found this to be one of the most critical 
elements of getting the recruitment to work. We have often sought to choose partners for this role 
early at the proposal stage, partners who are credible with the target populations and may already 
have trusted relationships. The best local contacts can also leverage their relationships in the 
community to gain the attention of target audiences. An outsider cold-calling a school or sending 
emails to people they have not met is less likely to receive a response than someone local who 
can work through his or her professional network. In SimCalc, the team partnered with regional 
Educational Service Centers (ESCs) in Texas to do recruitment and provide the professional 
development workshops. ESCs had two advantages as recruitment agents: They had 
established relationships with teachers in their region (and thus where not viewed with suspicion) 
and were already credible service providers to the schools. Each ESC was given a contract 
with clear recruitment quotas. This strategy worked very well. In Maine, representatives from 
the University of Maine did some of the recruitment on the basis of relationships they had with 
educational leaders in the state. The intervention developer hired as the other main recruiter an 
experienced local teacher who was also a veteran user of the intervention; he spoke to teachers 
with empathy for their jobs and with conviction about the benefits of the program. In West 
Virginia, for the Reasoning Mind study, the state Department of Education was willing to send all 
schools an introductory email about the study, which generated interest from some principals. 

A related consideration is how to take advantage of existing communities, organizations, and 
events in the target population. Getting the word out through existing networks and at existing 
events is much easier than through new ones. Thus, while planning a recruitment process, we 
recommend considering where target school leaders and teachers naturally congregate and 
how they typically get news. For example, in Texas the team identified TASM (Texas Association 
of Supervisors of Mathematics) as an important organization with a key event we could 
attend. In Maine, we checked the calendars of similar gatherings of school principals and of 
mathematics instructional leaders. School news often makes the local papers in Maine and is 
well read (this is less true in Texas); an early article in the Bangor Times was helpful in getting 
the word out to teachers and principals. The Project-Based Inquiry Science Efficacy study team 
created brochures that they handed out at national and regional professional conferences, such 
as those sponsored by the National Science Teachers Association. 

Finally, incentives can be considered for teachers or principals to help recruit their peers. Given 
the two-cohort design of the ASSISTments project, the team encouraged treatment principals 
and teachers in the first cohort to share their experience with potential recruits for the second 
cohort. Teachers and principals who did so received a referral bonus for each school that 
applied to the study upon their recommendation. 
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The framing of a research study to convince peer reviewers to rate a proposal highly has 
almost nothing to do with the best framing to attract recruits. Considerable thought must 
be given to the messaging for research study participants before recruitment begins and as 
recruitment proceeds. A metaphor for this would be shift from a “research” orientation to a 
“customer service” orientation, regarding the participants as the customers to be served. This 
is also a time when the project leader should consider if any new kind of staffing or capabilities 
you need for this— e.g. graphic designer, practitioners to help shape the messaging, or 
someone with marketing expertise. 


What Should Be in the Messages 

In our experience, the recruitment messages must convey five points. First, as with all selling, 
people are motivated more to relieve pain than to make their world a little bit nicer. Thus, it 
helps to frame the study in terms of teachers’ perceived problems and important, immediate 
needs. The message should be clear about how the study will address these problems 
and needs. For SimCalc, the perceived problem was the difficulty of teaching particularly 
challenging and important curricular topics well. For ASSISTments, it was making homework 
less burdensome and more valuable. In these two recruitment efforts, the teams tended to 
avoid broad pronouncements (such as national needs to close achievement gaps) and instead 
pinpointed specific local needs and concerns. 

Second, the message also must give a clear image, in jargon-free language, of what the 
intervention will do for the participants. For example, the team characterized participation 
in ASSISTments research as “making homework more productive and effective” rather than 
“trying out an intelligent tutoring system.” 

Third, potential costs and benefits should be summarized in the message. ASSISTments is 
free, and the Reasoning Mind curriculum is provided to participating schools at no charge 
during the study. Consider also nonmonetary benefits of participation, such as the provision of 
high-quality professional development or making the analysis results available to all (e.g., in the 
case of ASSISTments, best practices for making homework more effective). 
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We have tended to identify specific benefits at several levels: for the individual teacher, for the 
students and their schools, and for others in the state and country. For example, teachers will 
receive professional development and tools to support formative assessment or differentiated 
instruction; students will receive better support for learning; schools will gain insight into what 
similar but higher performing schools use as best practices; the state and nation will benefit 
from good research on homework (a somewhat controversial and pervasive issue). Ancillary 
motivation can arise when participants feel they are playing a role in important research funded 
by well-known agencies, such as the National Science Foundation, Department of Education, 
or the Bill & Melinda Gates Foundation. 

Fourth, requirements to participate (e.g., only seventh-grade mathematics teachers) and 
expectations of participants (e.g., teachers are expected to teach seventh grade for 2 years in a 
row during the study) should be succinctly stated. 

Finally, messages should be designed to address the typical concerns of participants. 
Additional concerns can include: 

• Alignment and fit with local mandates and context 

• Opportunities to customize and adapt the intervention 

• Excepted ease or difficulty of implementation 

• Continued availability of the product after the study ends 

• Availability of support during the study 

• The opportunity cost associated with potentially ending up in the control condition (or 
whichever condition is considered less preferable) 

• Privacy and protection of teacher and student 

• Ability to drop out of the study later (all our studies have had voluntary participation, as is 
required by IRB). 

Communicating about Random Assignment 

In our experience, planning how recruiters will talk about random assignment deserves 
discussion ahead of time. We strive for explanations that make both conditions equally 
desirable, to the extent possible. Advantages of a delayed condition, for example, include the 
opportunity to use the program after it has been better tested and further refined. Advantages 
of a control condition can include the need to do less work, the opportunity to share 
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established and potentially highly effective practices with researchers (the control condition, 
after all, may work better), and the value of the data about the school that will become available 
through participation in the study. We also emphasize the importance of random assignment 
as a scientific procedure and the contribution a participant makes by participating. We also 
seek to make it clear that suitable incentives are provided in both conditions, so neither is more 
financially rewarding. 


Message Formats 

In all our studies, preparing consistent and coherent messages in a variety of formats has 
been useful. One-page flyers, postcards, or short brochures are mailed to teachers, principals, 
or district administrators or handed out at meetings. Face-to-face presentations are usually 
an essential aspect of recruiting, often with demonstrations. A simple website can serve as a 
central place for sharing information about the study, collecting email addresses of potential 
recruits, and providing application forms for people who are interested in participating to 
download and submit. Sometimes, a short video about the study can also be very useful 
in getting a consistent message out to the target audience. Along with any of these bulk 
messaging techniques, following up in person with potential recruits is always important. 

Formatting messages as templates has advantages for personalization and consistency. 
Recruiters can personalize templates without needing to write their own message. Using 
templates keeps messages clear and consistent and gives every recruiter an equal opportunity 
to make a good first contact. 

As many recruiters will have to answer these and additional questions, preparing a “frequently 
asked question” list can be very valuable and helpful. 


Applying to Participate 

Finally, in many of the studies referenced in this paper, we have required recruits to go through 
the formality of applying to the study and being accepted (even if, in reality, almost all will 
be accepted). The main reason for this is to develop commitment to the project. We design 
application forms so that thought is required to complete them, and we require signatures 
from both administrators and teachers as an indication of commitment. (The signatures are 
nonbinding because participants are all volunteers who can drop out at will.) 

An example of how we made sure we were recruiting well-informed districts is from a study 
of Common Core-aligned professional development on teaching writing (See the College- 
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Ready Writers Program in the appendix). Local professional development providers recruited 
districts willing to partner with them for 2 years of professional development. As part of the 
initial recruitment process, SRI required a senior district administrator to sign a form that 
explained the program and requirements of each condition, thereby indicating a willingness 
to be randomly assigned to either condition. Senior researchers then scheduled a 20-minute 
telephone call with the administrator in each potential district to go over the information, 
explain the random assignment process, and enable the administrator to ask questions and 
raise concerns about the program or the assignment conditions. The formal sign-off coupled 
with the open conversation provided leverage later on in the study when districts considered 
withdrawing. This frank, up-front discussion of randomization has been critical to supporting 
the conduct of the RCT. 

In the SimCalc project, we required teachers to have their school leaders co-sign their 
application to make sure they have talked together about the study and agree. For 
ASSISTments, we refined this to produce an application package, with separate forms for 
the district superintendent, the principal, and each seventh-grade teacher in the school. 

The application lays out expectations of participants, as well as the financial incentives, so 
teachers can confirm they are aware of what they are signing up for. Further, the application 
collects some further basic data about the school and the teachers to ensure the school 
meets recruitment requirements, such as availability of technology and policies that align with 
the study requirements (e.g., for a homework study, that the school requires students to do 
homework) or whether the characteristics of the school or teacher meet the sampling criteria. 

An application is either submitted or not, making a school’s participation status clear to both 
the school and the researchers, avoiding miscommunication and disappointment on either side. 
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Running a Recruitment Campaign 




Recruitment can best be understood as a campaign. As in an election campaign, the goal is 
challenging, the process unfolds as unforeseen obstacles arise, and many people must work 
together in a coordinated manner to make it happen. 

At the center of the campaign, we have typically held weekly conference calls among all the 
key stakeholders— the frontline recruiters, the intervention provider, and the researchers (who 
sometimes include partners beyond SRI). This is necessary to handle emergent issues and to keep 
the work appropriately coordinated. Recruiters often need clarification on what they can say or 
cannot say to teachers or schools and help in addressing questions from potential recruits. Training 
frontline recruiters, especially if all is done by phone/email, helps with consistency and clarifications. 
The intervention provider can help clarify the intervention and its implementation in school and may 
need to make additional information or support materials available as issues emerge. Researchers 
need to respond flexibly to recruitment challenges, considering what best preserves the overall 
intent of the research given that perfection is out of reach. A crucial component of the weekly 
coordination meeting therefore is to share information across the team on what is working, what 
obstacles have been encountered, and what adjustments are recommended. 

A successful campaign, in our experience, operates on many simultaneous fronts but does 
not overwhelm the target participants. One front may be broadcast-oriented, seeking visibility 
through various media outlets, mention in newsletters from networks that schools participate 
in, and presence at events and conferences. Another front may involve substantial time on the 
ground, for example, driving from district to district and meeting one-on-one with key decision 
makers. Some activities may be led by the intervention developer (who may already have access 
to some target populations); yet particular activities (such as reassuring district officials about the 
integrity of the research) are best done by the research team. Determining what the important 
work currently is and coordinating assignment of that work among the project staff is a key 
function of the weekly conference calls. In addition, once communication is established with 
a potential recruit, providing a single point of contact for the study is helpful. This streamlines 
communications and avoids confusion with messaging. 

An essential tool is a shared database of all schools (or potentially each teacher) in the target 
participant population. The database should contain relevant demographic information on each 
school, such as state standardized test performance, AYP (Adequate Yearly Progress), Title 1 school 
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or not, and enrollment size. Early in the process, this database can help recruiters focus on particular 
groups of schools. For example, in the SimCalc study, the team gave each ESC a prioritized list of 
schools in its region to help it focus on recruiting the diverse population we were seeking. In the 
ASSISTments study, the team examined the database repeatedly to see what kinds of schools might 
be under- or over-represented in the recruitment achievements to date and to retarget efforts to 
obtain the desired variability in the sample. As the recruitment process goes on, the database can be 
used to track the demographic and educational data of recruited schools that should be reviewed 
against the sampling plan on a regular basis. 

We also have found it helpful to follow a sales pipeline metaphor in recruitment by tracking schools 
through multiple levels of engagement. At the beginning of the pipeline, we might include schools 
or teachers who have made any sort of favorable comment or expressed interest in the intervention 
by any means, regardless of how slight. Then we move on to focus on principals or teachers who 
have spoken personally to a recruiter and discussed the project or have attended a demonstration or 
presentation about the project. We set up a database to track schools or teachers who have indicated 
intent to apply (or not to apply) and their application status to date. The team reviewed regularly the 
numbers of applications intended and numbers of conversations needed, given the expected hit rate. 
Tracking this pipeline from leads to commitments can help recruiters to prioritize their work to both 
reach out broadly enough and to close the deal with the most promising prospects. In Reasoning 
Mind and the AP Science Impact Study recruitment efforts, the team set up explicit milestones and 
used Salesforce to literally implement a sales pipeline model, along with sophisticated lead tracking. 

We have found that running successful campaigns requires flexibility and adaptability. Some plans may 
not work, so other plans must be made. The success of particular plans may vary by school. Some 
schools may be immediately interested and sign up five classrooms, whereas other schools may require 
many communications and offer just one classroom to participate. A plan that worked on a previous 
project may not work for a different one. In such cases, we have had to increase incentives or add 
features like the school-referral bonus. As the team discusses what is working and what is not working, 
changes to the messaging may be needed. Further, the team may learn that certain conditions in 
districts or schools are favorable or unfavorable to recruitment and thus may retarget its efforts. 

Keeping team spirit and motivation up is also crucial during the campaign. Recruitment can often be 
dispiriting, and every recruitment effort goes through times when hopes of meeting the target recede. 
Thus, we periodically remind everyone on the team of why recruitment is so important— for example, 
that the developer of the program wants the best chance of measuring an effect and that requires an 
adequate sample size. It is also important to celebrate incremental victories, to listen to war stories 
of problems in the field, and to respond to challenges quickly. Resources often must be allocated 
quickly (e.g., to host a booth at an event), and prompt decision-making keeps everyone focused on 
the action rather than waiting for decisions to be made. 
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After Recruitment 


Although the whole team needs a celebration at the end of a successful recruitment campaign, 
the effort is hardly over. Random assignment must be done promptly and participants need to be 
notified. 

Of primary concern to teachers is prompt notification of which condition they have been assigned 
to and entering obligations for summer professional development workshops onto their calendars 
before school year ends. In the ASSISTments and Reasoning Mind studies, we randomly assigned 
schools to condition in batches (after matching schools on key student demographics and prior 
performance) so that early recruits would not have to wait for late recruits. In matching participants 
before random assignment, the batches should be large enough to allow for reasonable matching of 
participant pairs; in our practice, batches of 8-1 0 schools have been sufficient. 

Schools or teachers may change their minds and drop out of the study upon learning about the 
results of assignment to conditions or even during the study. This is another critical stage when the 
“customer service” orientation is important in order to guard against attrition. Following through 
during the intake process of the study is important to prevent attrition. Not only is a warm welcome 
important, but ongoing communication (especially among business-as-usual participants) is also 
key to prevent attrition. Keeping teachers engaged with reminders and notes of appreciation helps 
maintain a connection and sense of commitment to remain involved. We often talk about the 
Two Rs, namely, rapport and relationships. Building and maintaining the Two Rs are essential in 
preventing attrition, especially in longer studies where the contexts under which participants joined 
(e.g., district or school leadership, policies that the intervention is designed to address) may change 
over the course of the study. To build rapport and relationships in the ASSISTments study, team 
members made phone calls individually to all schools that submitted applications, informed them 
of the assignment results, and made sure they were still on board. One school decided to withdraw 
from the study when it was notified that it had been assigned to the treatment condition, the 
representative saying, “Too much is going on in school.” 

Luckily, in ASSISTments, our recruitment pool was large enough to accommodate the last-minute 
withdrawal without needing to recruit more schools, but the situation might be different in other 
studies. The example does highlight the challenge of crafting messages that are simultaneously 
attractive to potential participants yet force them to think through what it would actually be like to be 
included the study. 
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Conclusion 


Recruiting for an RCT is particularly intense and challenging. Participants must be eager to 
implement the target intervention and yet equally willing not to implement it, depending on random 
assignment. Participants must be willing to give up control of an aspect of their school or classroom. 
Participation often involves obtaining coordinated commitments from multiple levels of a school 
system: teachers, principals, and district leaders. 

For successful recruitment, we have found it necessary to plan and execute a disciplined process 
over a considerable length of time, with involvement of an extended research and recruitment team. 
Indeed, a campaign is an apt metaphor for the process. 

From our experiences with recruiting for many projects, we have found it useful to conceptualize 
recruitment in phases, corresponding to (a) study design, (b) intervention packaging, (c) planning the 
recruitment process, (d) designing recruitment messages, (e) running the recruitment campaign, and 
(f) following up after recruitment. Each phase has a detailed set of considerations appropriate to its 
essential work. 

Given that recruitment is the foundation of a successful RCT, careful attention to the process from the 
time the study is conceived until the last pieces of data are in is critical. Consequently, we advocate 
for more detailed attention to recruitment in research proposals, in the work of peer review panels 
and pre-award negotiations, and in the work of advisory committees. Our experience indicates that a 
well-formulated recruitment plan clearly fits the research design; explains how the intervention will be 
packaged to reduce the burden on participants; describes the overall timeline, staffing, and project 
management structure for the campaign; gives examples of how messages will be communicated to 
potential recruits; discusses strategies for handling contingencies that emerge during the campaign; 
and contains plans for how applicants will be welcomed into the study and retained as participants. 
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Appendix: Overview of SRI Education RCTs 



In this appendix, we included structured abstracts of the RCTs mentioned in the white paper. Each 
abstract contains the following information: 

• Program description 

• Main research question/hypothesis 

• Setting— description of the research location and partners (if applicable) 

• Population -description of participants in the research: who, how many, key features/characteristics 

• Study design— level of randomization, clustering 

• Data collection methods— key activities and main measures 

• Funder 

• Periods of performance (years) 

• Subject area: Math, ELA, Science, Early Learning, Pre-service Teachers, etc.. 

List of Structured Abstracts for Included RCTs: 


Evaluating the Impact of Professional Development to Meet Challenging Writing Standards in High-Need 
Elementary Schools on page 27 

An Efficacy Study of Online Mathematics Homework Support: An Evaluation of the ASSISTments Formative 
Assessment and Tutoring Platform on page 27 

Efficacy of an Integrated Digital Elementary School Mathematics Curriculum on page 28 
Scaling Up SimCalc on page 29 

Ready To Learn (RTL) Prekindergarten Transmedia Mathematics Study on page 29 
Impact Evaluation of the College-Ready Writers Program on page 30 
National Evaluation of Writing Project School Partnerships on page 31 
Project-Based Inquiry Science (PBIS) Efficacy Study on page 31 
Impact Evaluation of the Pathway to Academic Success Project on page 32 
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Evaluating the Impact of Professional Development to Meet 
Challenging Writing Standards in High-Need Elementary Schools 

The National Writing Project supported its local Writing Project sites in offering intensive teacher 
professional development on the Common Core State Standards during the 2012-13 academic 
year. The professional development was for teachers in grades 3 through 5 in high-need schools and 
emphasized opinion/argument writing. The study sought to estimate the impact of the professional 
development on teacher instructional practices and student writing. 

The study was conducted in 12 Local Writing Project sites in 10 states across the country. Each 
Local Writing Project site recruited one to three pairs of schools. SRI randomized schools within each 
site, resulting in 21 program and 21 control schools. 

SRI administered on-demand writing prompts to all third-, fourth-, and fifth-grade students in 
these 42 schools in fall 2012 and spring 2013 to estimate program impacts on their writing. SRI 
administered surveys to approximately 400 third-, fourth-, and fifth-grade teachers in fall and 
spring to learn about their instructional practices and experiences with professional development. 
Additionally, SRI collected data on teacher participation in professional development and interviewed 
teachers, school administrators, and professional development providers to learn how individuals 
experienced the program and how variations in school context influenced participants’ uptake of 
ideas from the professional development. 

The study was funded by the National Writing Project under its Supporting Effective Educator 
Development (SEED) grant from the U.S. Department of Education. 

An Efficacy Study of Online Mathematics Homework Support: 
An Evaluation of the ASSISTments Formative Assessment and 
Tutoring Platform 

ASSISTments is a web-based system that delivers mathematics homework from textbooks to 
students online, gives students instant feedback on their responses, and provides teachers with 
reports about homework. The primary research question of Online Math Homework Efficacy Project 
is whether online homework can lead to increases in seventh-grade students’ mathematics scores 
on Common Core assessments. Further, the study seeks to understand how such impacts are 
produced, which could be from any of (a) greater homework completion by students, (b) more 
feedback on homework to students, or (c) feedback to teachers about homework that supports 
adaptive teaching routines in the classroom. 

The state of Maine was selected for this project because it gives a laptop to every student to take 
home, so students have the equipment to do math homework on a computer. A total of 1 1 9 
teachers from 49 middle schools are participating in two cohorts; the second cohort started a year 
later than the first. The schools were sampled from across Maine, with the intention of representing 
the variation in school size and prior achievement scores that naturally occur in the state. 
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Schools in each cohort were randomly assigned to either use ASSISTments for online homework or 
continue in their business-as-usual homework practices. All seventh-grade teachers in a school are 
therefore in the same condition, and all the seventh-grade students they teach are in the study. All 
teachers in the ASSISTments program participate in teacher professional development to introduce 
them to the technology, support their use of it for online homework, and encourage them to engage 
in adaptive homework review practices based on the data they get from ASSISTments. 

The primary outcome measure is the Terra Nova assessment, Common Core edition. Additional 
measures include interviews with principals and teachers, observations of classroom practices, 
teacher homework logs, and a unit test on one key topic in seventh-grade mathematics. The 
ASSISTments system also collects data automatically about homework assignments, student 
performance on homework, and teacher use of reports. 

Efficacy of an Integrated Digital Elementary School 
Mathematics Curriculum 

Reasoning Mind offers a complete technology-based curriculum for fifth-grade mathematics and 
includes substantial teacher professional development and implementation support. The curriculum 
is aligned with the Common Core. The technology is intended to motivate student engagement, give 
students immediate feedback, adapt the pace and pathways of instruction to student needs, and give 
teachers feedback they can use to target their teaching activities to student needs. The study seeks 
to determine the impact of a full-year blended curriculum for fifth-grade mathematics on the Smarter 
Balanced assessments relative to business-as-usual curricula and supplementary digital products. 

The state of West Virginia was selected for this study because it has invested in broadband and 
technology for its schools but has low mathematics achievement, and technology offers important 
hope to rural communities in improving mathematics achievement. Within the state, a population 
was recruited both to capture the rural character of much of the state and to sample midsize 
population centers, which are found in both rural and nonrural states. 

One cohort of 50 schools was selected. Schools were randomly assigned to either use Reasoning 
Mind as their fifth-grade core curriculum or continue with business-as-usual curricula and 
technologies. To balance incentives, schools in the business-as-usual condition will have a different 
Reasoning Mind product for use in second grade. Because the second-grade students will not reach 
fifth grade during the study, the threat of contamination from this procedure is low, and ensuring that 
every school gets a version of Reasoning Mind for one grade is a substantial incentive. 

The primary outcome measure is the Smarter Balanced assessment, as required by West Virginia. 
Additional measures are teacher interviews, classroom observations, implementation fidelity reports, 
measures of teacher knowledge, and automatically collected system usage data. 
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Scaling Up SimCalc 

SimCalc is an integration of mathematics software, paper curriculum modules, and teacher 
professional development aimed at replacing how teachers instruct students on two to three topics 
in seventh- and eighth-grade mathematics. SimCalc features multiple representations, including 
motion and graphs, that support student conceptual understanding of challenging mathematical 
ideas, such as rate and proportion. The primary research question in the SimCalc studies was 
whether this integration increases student performance on advanced algebra-relevant concepts and 
skills as well as performance on more basic items typical of the Texas statewide assessment. 

Texas was selected for this study because of its diverse population and its stable and well- 
established system of standards and accountability. Further, four regions of Texas were selected to 
represent more urban and cosmopolitan areas, more rural areas, and areas near the Mexican border. 
A total of 95 teachers were recruited for the seventh-grade study, and 63 teachers were recruited for 
the eighth-grade study through the Educational Service Centers, which ordinarily provide teacher 
professional development in these regions. 

Schools were randomized to condition, but most schools had only one participating teacher. The 
seventh-grade study featured a delayed treatment design, whereby the control group was offered 
SimCalc after a 1 -year delay. The eighth-grade study was a simple SimCalc or business-as-usual 
comparison. 

Measures included a researcher-designed assessment with scales related to algebra and to the 
Texas state test, a test of teachers’ mathematical knowledge for teaching, interviews, observations, 
and teacher self-report logs. 

Ready To Learn (RTL) Prekindergarten Transmedia 
Mathematics Study 

The RTL Prekindergarten Transmedia Mathematics Study is a principal part of SRI and Education 
Development Center, Inc.’s summative evaluation of Ready To Learn. RTL is a U.S. Department of 
Education-supported initiative that aims, in partnership with the Corporation for Public Broadcasting 
and PBS, to develop media-rich learning resources for young children from economically 
disadvantaged communities. 

The goal of this randomized controlled trial was to evaluate the impact of the RTL transmedia 
math supplement on young children’s mathematics learning, its influence on teacher attitudes 
and beliefs about early mathematics education and the use of technology/media to support 
mathematics learning, and the successes and barriers teachers encountered in implementing 
the curriculum supplement. 

The study took place in 92 classrooms (46 in New York City and 46 in the San Francisco Bay 
Area) in preschool agencies and centers serving children primarily from low-income households. 
Classrooms were randomly assigned to one of three conditions— the PBS KIDS transmedia math 
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supplement, technology & media, or business as usual— and data from classroom observations, 
child math assessments, teacher surveys, and weekly teacher and coach logs were collected. A total 
of 1 57 teachers participated in the study, and 966 children ages 4 to 5 were randomly selected for 
assessment, approximately 10 children from each classroom. 

The primary child outcome measures were the Supplement-Based Assessment of Mathematics 
(SBA), an assessment designed during the 2012 pilot study to assess children’s understanding of 
the concepts and activities in the PBS KIDS Transmedia Math Supplement, and the short version of 
the Research-based Early Mathematics Assessment (REMA), a standardized measure of preschool 
children’s early numeracy and geometry skills. 

Impact Evaluation of the College-Ready Writers Program 

SRI is investigating how professional development aligned with college- and career-ready standards 
affects writing instruction and in turn student writing performance. The professional development 
provided by local sites of the National Writing Project focuses on using evidence from text to 
articulate and support written arguments. In participating districts, 7th- through lOth-grade English 
language arts (ELA) teachers will receive at least 90 hours of professional development over 2 years. 
The study will estimate program impacts on writing instruction in 7th through 10th grade and the 
quality of student argument writing. 

SRI is conducting the study in 44 rural school districts, which are partnered with 1 2 Local Writing 
Project sites, located in 1 0 states. SRI randomized the districts to receive the 2-year College-Ready 
Writers Program beginning in the 201 3-14 school year (treatment group) or to receive the first year of 
the program beginning in the 201 5-1 6 school year (control group). 

SRI is collecting longitudinal data in the form of 

• Surveys of about 1 ,000 ELA, social studies/history, and science teachers 

• Weeklong instructional logs from more than 300 ELA teachers 

• Student writing prompts administered over 2 days to more than 20,000 students 

• Records of teacher participation in professional development on writing instruction. 

Additionally, SRI is collecting qualitative data through telephone and on-site interviews with district 
leaders and teachers, as well as observation of professional development and classrooms. 

This study will contribute to understanding of what districts can do to improve student writing and 
how to design effective teacher professional development. The project is funded by the National 
Writing Project through an Investing in Innovation grant from the U.S. Department of Education. 
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National Evaluation of Writing Project School Partnerships 

SRI investigated 3-year partnerships between schools and local sites of the National Writing Project 
that gave schools access to writing professional development. The schools worked with Local 
Writing Project sites to co-plan the professional development that school leaders thought best met 
their needs. Local Writing Project sites then provided the professional development. The study 
sought to describe the professional development and estimate its impacts on teachers and students. 
A total of 39 schools in 1 0 states that served middle grades (i.e., included seventh and eighth grade) 
participated in the study. 

SRI randomized 40 schools to either have 1 year of planning followed by 3 years of professional 
development or maintain a business-as-usual control condition. During the 4-year study, SRI used 
multiple data collection strategies including teacher surveys, teacher logs, teacher assignments, 
student work, on-demand writing prompts, interviews, and document review to collect data on 
partnership implementation and assess impacts on teachers and students. The project was funded 
by the National Writing Project. 

Project-Based Inquiry Science (PBIS) Efficacy Study 

SRI is studying the benefits of a new middle school science curriculum based on contemporary 
research in science teaching and learning that aligns with the new standards (e.g. the Next 
Generation Science Standards), provides comprehensive instructional support for teachers, and 
incorporates the latest research on student learning. 

The commercially available curriculum, Project-Based Inquiry Science (PBIS), focuses on standards- 
based content and uses project-based inquiry units to help students learn. In PBIS classrooms, students 
investigate as scientists would— through observations, designing and carrying out experiments, 
building and using models, reading about the science they are investigating, constructing explanations, 
and so forth— but in ways that are appropriate and meaningful for them and with carefully structured 
support. With an emphasis on important science content and integrating scientific practices, the PBIS 
curriculum’s design matches well with the new directions in science education. 

Funded by the National Science Foundation, SRI is conducting the efficacy study on the benefits 
of PBIS for supporting science teaching and students’ science learning. A unique feature of the 
study design is an analytic focus on the conditions needed to implement the curriculum in ways 
that improve student learning in light of the new Framework for K-1 2 Science Education and Next 
Generation Science Standards . The study is being performed in collaboration with Michigan State 
University and the University of Colorado, Boulder. 

The study began in August 2012 and involves approximately 1 00 sixth-grade science teachers and 
more than 3,000 students from 42 middle schools. This study is the first to examine teachers’ and 
students’ use of science curriculum materials with measures that are aligned with the Framework for 
K-1 2 Science Education. Moreover, it is one of a handful of rigorously designed impact studies that 
also examine implementation and effects on teaching practice systematically. 
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Impact Evaluation of the Pathway to Academic Success Project 

SRI is studying the impact on the Pathway Project on English learners’ academic literacy skills. 

The study will test whether English learners in grades 7-12 whose teachers receive professional 
development and instructional materials from the Pathway Project are better able to analyze the 
text and to write a well-evidenced essay based on their analysis. The professional development and 
instructional materials that constitute the Pathway to Academic Success Project are being provided 
by four National Writing Project sites. The study is being conducted in seven districts in southern 
California located in the Writing Project sites’ service areas, which serve high populations of English 
learners. 

After students are assigned to classes in the fall of 2014, SRI will randomly assign teachers within 
schools to receive Pathway during 201 4-1 5 and 201 5-1 6 or 201 6-1 7. Professional development 
observations and attendance records will enable researchers to assess implementation fidelity. 
Researchers will assess students’ writing using the holistic score from the National Writing Project’s 
Analytic Writing Continuum in the fall of 2014, spring of 2015 and spring of 2016 to estimate 
program impacts on student writing. Additional student and teacher surveys will enable researchers 
to estimate impacts on teacher practice. This experiment, funded as a validation grant under the 
federal Investing in Innovation program runs from 2014-2018. 
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